Skip to content

feat: support GLM-5.3-Flash inference - #1525

Open
sufubao wants to merge 29 commits into
ModelTC:mainfrom
sufubao:support-glm-5-3-flash
Open

feat: support GLM-5.3-Flash inference#1525
sufubao wants to merge 29 commits into
ModelTC:mainfrom
sufubao:support-glm-5-3-flash

Conversation

@sufubao

@sufubao sufubao commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

最终镜像

registry.ms-sc-01.maoshanwangtech.com/ms-ccr/lightllm@sha256:0260e9884e46de899f4b845aa3796d0603b7b6ee7593c1fded35b7cee4462169

H100 启动命令

IMAGE="registry.ms-sc-01.maoshanwangtech.com/ms-ccr/lightllm@sha256:0260e9884e46de899f4b845aa3796d0603b7b6ee7593c1fded35b7cee4462169"

sudo docker pull "$IMAGE"
sudo docker run -d \
  --name glm53-lightllm-16k256-kpool \
  --restart unless-stopped \
  --network host --ipc host --gpus all \
  -e LIGHTLLM_ENABLE_KPOOL_DECODE_FASTPATH=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --ulimit memlock=-1 --ulimit nofile=1048576:1048576 \
  -v /home/devsft/models/GLM-5.3-Flash:/model:ro \
  -v /home/devsft/cache-glm53-lightllm:/root/.cache \
  -v /home/devsft/cache-glm53-triton:/root/.triton \
  -v /home/devsft/cache-glm53-deep-gemm:/root/.deep_gemm \
  "$IMAGE"

1M 上下文启动命令

1M 上下文使用已完成精确 1,000,000-token needle 验证的专用镜像:

registry.ms-sc-01.maoshanwangtech.com/ms-ccr/lightllm:glm-5.3-flash-260901-1m

不可变 Digest:sha256:3d07b3e9964cae15001e8136f1d19bd0b655df58488546e32f1dd15ffc9dbab7

IMAGE_1M="registry.ms-sc-01.maoshanwangtech.com/ms-ccr/lightllm:glm-5.3-flash-260901-1m"

sudo docker pull "$IMAGE_1M"
sudo docker run -d \
  --name glm53-lightllm-vl-1m-tp8-c256 \
  --restart unless-stopped \
  --network host --ipc host --gpus all \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  --ulimit memlock=-1 --ulimit nofile=1048576:1048576 \
  -v /home/devsft/models/GLM-5.3-Flash:/model:ro \
  -v /home/devsft/cache-glm53-lightllm:/root/.cache \
  -v /home/devsft/cache-glm53-triton:/root/.triton \
  -v /home/devsft/cache-glm53-deep-gemm:/root/.deep_gemm \
  "$IMAGE_1M" \
  /opt/sglang/bin/python -m lightllm.server.api_server \
  --model_dir /model \
  --model_name glm-5.3-flash \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8002 \
  --httpserver_workers 16 \
  --mem_fraction .90 \
  --max_total_token_num 1048612 \
  --running_max_req_size 256 \
  --max_req_total_len 1048576 \
  --batch_max_tokens 16384 \
  --chunked_prefill_size 1024 \
  --linear_att_ssm_data_type bfloat16 \
  --linear_att_cache_size 256 \
  --graph_max_batch_size 256 \
  --graph_split_batch_size 8 \
  --graph_grow_step_size 16 \
  --graph_max_len_in_batch 2048 \
  --disable_flashinfer_allreduce \
  --enable_fused_shared_experts \
  --disable_dynamic_prompt_cache \
  --disable_aggressive_schedule \
  --router_max_wait_tokens 64 \
  --max_image_pixels 6272000 \
  --max_image_token_count 8000 \
  --visual_tp 1 \
  --visual_dp 8 \
  --visual_infer_batch_size 8 \
  --cache_capacity 64 \
  --schedule_time_interval 0.001 \
  --prefill_coalesce_interval 0.5 \
  --reasoning_parser glm45 \
  --tool_call_parser glm47

16K 输入 / 256 输出

同一台 8×H100 80GB、同一 FP8 checkpoint、TP8、32K batch token、关闭 prompt cache;总吞吐单位为 tok/s。

并发 LightLLM vLLM LightLLM / vLLM
1 5,373.86 6,542.89 82.13%
8 16,444.11 17,635.33 93.25%
16 20,669.61 25,246.99 81.87%
64 25,614.05 31,485.12 81.35%
128 25,588.22 27,988.09 91.43%
256 25,623.93 31,206.94 82.11%

精度:GSM8K 99/100;精确 16,384-token needle PASS,找回 ZEBRA-4821

sufubao and others added 22 commits August 28, 2026 19:10
# Conflicts:
#	lightllm/common/basemodel/basemodel.py
#	lightllm/common/basemodel/batch_objs.py
#	lightllm/common/basemodel/triton_kernel/post_process/vocab_parallel_greedy.py
#	lightllm/models/llama/layer_infer/post_layer_infer.py
#	lightllm/server/router/model_infer/mtp_speculative/proposers/eagle_with_att.py
#	unit_tests/common/basemodel/test_model_output.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant